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numericai  method.  Combining  this  new  theon/  with  some  convex  programming  based  pruning,  we 
demonstrated  solution  of  a  particular  class  on  nonlinear  problems  over  six-dimensional  space.  Standard 
solution  methods  would  take  computational  time  on  the  order  of  decades  to  solve  such  a  problem, 
whereas  we  were  able  to  obtain  a  solution  in  under  an  hour  on  a  desktop  machine  for  the  example 
problem. 

We  also  investigated  sensing  UAV  tasking  algorithms.  We  demonstrated  that  the  correct  criterion  for 
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efficiently  with  idempotent  methods.  This  was  unexpected,  as  previously  it  was  believed  that  one 
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was  found  to  lie  in  the  idempotent  distributive  property. 
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Introduction 


Multiple  breakthroughs  were  made  during  the  period  of  the  effort.  We  divide 
them  up  according  to  the  two  main  areas. 


2  Curse-of-Dimensionality-Free  Methods  for 
HJB  PDEs 

At  the  outset  of  the  project,  we  had  the  basic  algorithm  in  hand  for  a  curse- 
of-dimensionality-free  approach  for  a  class  of  first-order  HJB  PDEs  where  the 
Hamiltonian  is  written  (or  approximated)  as  a  pointwise  maximum  over  a 
finite  set  of  quadratic  forms.  Although  the  approach  is  generally  applicable 
to  such  problems,  for  the  purposes  of  demonstration  we  needed  to  choose 
a  particular  form.  We  chose  the  class  of  HJB  PDEs  corresponding  to  infi¬ 
nite  time-horizon,  average  cost  per  unit  time,  deterministic  control  problems. 
More  exactly,  we  consider  HJB  PDEs  of  the  form 

0  =  H(x,  VV)  Vrr  6  JRn  \  {0}, 

V"(0)  =  0, 

where 

H(x,p)  =  max  Hm(x,p), 
meM 

with  M.  =]1,  M[=  {1, 2  . . .  M}  and 

Hm(x,p)  =  \x'Dmx  +  \p'TTp  +  (. Amx)'p  +  {l?)'x  +  (U?)'p  +  am  Vm  €  M. 

Note  that  the  solution  is  not  a  piece- wise  linear-quadratic  function,  but  is 
fully  nonlinear. 

Although,  we  had  an  algorithm,  we  lacked  a  convergence  analysis  and 
error  bounds.  This  was  necessary  for  full  acceptance  of  the  approach.  The 
resulting  full  analysis  is  spread  across  [16]  and  [15].  In  order  to  give  a  sense 
of  the  results  we  note  that  at  each  step,  the  solution  is  represented  by  a 
pointwise  maximum  of  quadratics,  i.e.,  at  step  N,  the  approximation  is 

v"(x)  =  ©  Uf(x) 


2 


T 


with  each  represented  by  the  coefficients  of  the  quadratic,  (Qj^,x^,c^). 
At  the  next  step,  one  has  an  approximation  of  the  form 


VN*'(x)=  0  S? 

mGM 


©or 


(x) 


=  ©  ©  sr*  K]  (x)  =  ®  uf+i(x), 

m£Ai  kG)Ctf  kGfCtf+i 


where  each  semigroup,  S'™,  approximates  the  action  of  a  single  linear-quad¬ 
ratic  control  problem  associated  with  Hm.  The  computations  are  reduced 
to  analytical  (modulo  a  matrix  inverse)  operations  on  the  set  of  coefficients. 
The  computational  growth  in  space  dimension  is  only  n3,  as  opposed  to  Dn 
for  grid-based  methods,  where  D  is  the  number  of  grid-points  per  space 
dimension,  and  the  solution  is  only  obtained  over  a  finite  region  in  the  latter 
case.  The  time-step  is  r,  and  the  total  time  propagation  at  step  N  is  T  =  Nt. 
The  errors  go  to  zero  as  r  J,  0  and  T  — »  oo.  The  specific  error  bounds  axe  as 
follows.  For  an  error  on  the  order  of  e(l  +  |x|2)  over  all  of  Mn,  it  is  sufficient 
to  have  r  oc  e2  and  N  oc  e-3. 

This  elimination  of  the  curse-of-dimensionality  alone  was  significant,  but 
there  was  a  terrible  price  to  be  paid  in  the  above  basic  algorithm  in  terms  of 
what  we  call  the  curse-of-complexity.  In  particular,  Kfj  =  #/C/v  grows  like 
MN .  However,  it  was  quickly  noticed  that,  in  practice,  the  great  majority 
of  the  quadratics,  ,  typically  contributed  little  or  nothing  to  the  actual 
solution.  Various  pruning  strategies  were  applied  to  attenuate  this  complex¬ 
ity  growth  through  the  elimination  of  less-valuable  quadratics.  (Essentially, 
these  project  the  solution  onto  a  lower-dimensional  abstract  space  at  each 
step.)  However,  during  a  visit  with  Dr.  S.  Gaubert  at  INRIA,  we  determined 
that  certain  tools  from  convex  optimization  could  be  applied  to  radically  im¬ 
prove  this  pruning  process.  See  [20].  With  this  tool  in  hand  we  demonstrated 
solution  of  an  HJB  PDE  over  all  of  Ft6  with  M  =  6  in  roughly  45  minutes 
on  a  desktop  machine.  This  would  require  many  years  of  CPU  time  with 
standard  methods  (solving  only  over  a  relatively  large,  but  finite,  region  in 
the  latter  case).  Of  course,  one  cannot  depict  a  solution  over  6-dimensional 
space.  However,  some  data  along  planes  is  depicted  in  Figures  1-2. 

The  above  results  require  that  the  Hamiltonian  take  the  form  of  a  maxi¬ 
mum  of  quadratic  forms.  In  order  to  make  the  approach  amenable  to  general 
nonlinear  problems,  one  must  determine  a  means  for  approximating  a  gen¬ 
eral  (semiconvex)  Hamiltonian  as  a  maximum  of  quadratic  forms.  In  order  to 
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Figure  1:  x\  and  xi  partials  on  the  £1,2:2  plane 

do  this  sensibly,  one  needs  to  determine  a  mapping  from  the  approximation 
quality  of  the  Hamiltonian  to  the  resulting  error  in  the  solution.  This  will 
lead  us  to  appropriate  approximations  of  a  general  Hamiltonian.  In  [17],  [18], 
we  demonstrate  such  a  mapping. 

Value- Based  Sensor  Tasking 

The  correct  measure  for  sensor  UAV  tasking  is  the  expected  payoff  to  the 
warfighters  of  the  possible  observational  data  returns.  More  typical  ap¬ 
proaches  include  entropy-based  metrics,  but  these  are  not  correct  in  the 
sense  that  they  do  not  represent  the  actual  value  of  the  information  to  the 
warfighter.  For  example,  one  location  might  be  a  far  more  dangerous  po¬ 
tential  location  for  opposing  forces  to  be  firing  from  than  another.  In  [23], 
McEneaney  developed  an  object,  VT(q )  which  describes  the  minimax,  ex¬ 
pected  payoff  for  a  game  between  Blue  and  Red  at  time  t  as  a  function  of 
the  Blue  knowledge  of  the  system  state,  specified  as  probability  distribution 
q.  (It  is  assumed  that  Red  has  perfect  state  information.) 

This  object  may  be  used  to  determine  the  value  of  sensing  actions.  For 
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Figure  2:  Optimal  switching  and  backsubstitution  error  on  the  X\,X2  plane 


discussion  purposes,  suppose  a  simple,  decomposed  problem  is  given  as  fol¬ 
lows.  Suppose  Blue  will  choose  a  sensing  control  action,  ending  at  time, 
t  =  r  >  0,  immediately  followed  by  a  combat  action.  At  time,  t  =  0,  Blue 
knowledge  is  described  by  Qq.  Given  a  series  of  sensing  actions,  {u*}[=0,  there 
is  an  associated  set  of  possible  observations,  Note  that  the  yt 

are  random  variables  -  the  actual  observation  that  will  be  obtained  will  be 
corrupted  by  noise.  Given  such  a  set  of  observations,  one  may  update  the 
distribution  qo  to  qr  by  an  estimator  such  as  Bayes  rule.  Note  that  qT  is  a 
random  variable.  Let  the  resulting  expected  payoff  assuming  optimal  future 
troop  actions  be  denoted  by  V(t,  gr). 

Based  on  an  analysis  of  the  combat  actions,  in  the  case  where  the  opposing 
force  stays  fixed  in  their  possibly  hidden  urban  locations,  we  were  able  to 
show  that  this  value  will  always  take  the  form  of  a  pointwise  maximum  of 
linear  functionals  over  the  probability  simplex.  That  is,  one  has 

V(t,  q)  =  max  b\  ■  q. 

»€lr 

One  also  has  the  usual  dynamic  programming  result  that  for  t  €  {0, 1, . . .  r  — 
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V(t,q)  =  mxxEv{V(t  +  l,^’u(g))} 

ueu 

where  the  expectation  is  over  the  set  of  possible  observations. 

Very  interestingly,  we  found  that  if  V(t  +  1,  q)  takes  the  form  V(t+1,  q)  = 
maxjgxt+1  b\+1  ■  q  where  It+i  =  {1,2,...  /m}.  Then, 

V(t,q)  =  max  b\  ■  q 

where  Xt  =  {1, 2, . . .  7t),  It  =  Nu(It+ 1) N*, 

=  (i) 

yey 

where  ( u ,  {jj,})  =  M~l(i),  and  M.  is  a  one-to-one,  onto  mapping  from  U  x 
VNv(It+\)  — ♦  Tt  (i.e.,  an  indexing  of  U  x  VNy(It+ 1))-  Here,  D(Hy'u)  is  a 
Bayes  rule  updater;  the  linearity  is  a  result  of  an  expectation  operation. 
Using  this  technology,  we  began  solving  sensing  control  problems.  A  very 
simple  example  of  an  optimal  path  can  be  seen  in  the  right-hand  image  of 
Figure  3,  where  the  blue  path  indicates  movement  of  combat  ground  forces, 
and  the  green  path  indicates  the  optimal  tasking  path  of  a  single  supporting 
sensor  platform.  More  information  on  this  area  of  research  appears  in  [25], 
[26]. 

At  some  later  time,  we  realized  the  full  implication  of  this  result.  It  was 
that  one  essentially  had  a  max- plus  curse-of-dimensionality-free  approach 
to  a  stochastic  control  problem.  This  was  entirely  unexpected,  as  it  was 
previously  believed  that  max-plus  linearity  was  required  for  max-plus  curse- 
of-dimensionality-free  methods.  However,  in  fact,  wrhat  is  actually  required  is 
quite  a  bit  less.  We  are  now  exploring  idempotent  (max-plus  and  min-max) 
curse-of-dimensionality-free  methods  for  stochastic  control  problems  and  cer¬ 
tain  games.  This  is  the  beginning  of  a  new  branch  of  research  in  application 
of  idempotent  methods  to  control,  estimation,  games  and  Hamilton-Jacobi 
PDEs;  see  [11],  [12],  [13],  [14]. 
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